Fast Generation of a Sequence of Trained and Validated Feed-Forward Networks

In this paper, three approaches are presented for generating and validating sequences of different size neural nets. First, a growing method is given along with several weight initialization methods, and their properties. Then a one pass pruning method is presented which utilizes orthogonal least squares. Based upon this pruning approach, a onepass validation method is discussed. Finally, a training method that combines growing and pruning is described. In several examples, it is shown that the combination approach is superior to growing or pruning alone. Introduction According to the structural risk minimization (SRM) principle, a sequence of learning machines of increasing size should be produced, and trained as well as possible. The machine with the smallest validation error is the best compromise between the training error and the complexity of the network. If this principal is followed with neural nets, the final training error as a function of the number of hidden units, Ef(Nh), will be monotonically nonincreasing. Sequences of networks are produced though growing methods and pruning methods. In growing methods (Delashmit 2003), one can design a set of different size networks in an orderly fashion, each with one or more hidden units than the previous network (Fahlman et. al. 1990) (Chung & Lee 1995). A drawback of growing methods is that the network can get trapped in local minima and they are also sensitive to initial conditions. In pruning methods, a large network is trained and then less useful nodes or weights are removed (Hassibi et. al. 1993) (LeCun et.al. 1990) (Sakhnini, Manry, & Chandrasekaran 1999). Some pruning algorithms remove less useful hidden units using the Gram-Schmidt procedure as reported by Kaminsky and Strumillo (1997) for Radial Basis Functions and Maldonado et.al. (2003) Compilation copyright © 2006, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. for the multilayer perceptron (MLP). Unfortunately, if one large network is trained and pruned, the resulting error versus Nh curve is not minimal for smaller networks. In other words, it is possible, though unlikely, for each of the hidden units to be equally useful after training. In this paper, we investigate three approaches for generating and validating sequences of different size neural nets. First, a growing method is described, along with analyses of weight initialization methods. Then a pruning method is presented which requires only one pass through the training data. A one-pass validation method is presented, which is based upon pruning. Next, a method that combines growing and pruning is presented. Results show that this third approach overcomes the shortcomings of growing or pruning alone. Multilayer Perceptron Figure 1 depicts feed-forward MLP, having one hidden layer with Nh nonlinear units and an output layer with M linear units. For the p pattern, the j hidden unit’s net function and activation are